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Over the years, finite automata have been used effectively in the theory of infinite 
groups to represent rational subsets. This includes the important particular case of finitely 
generated subgroups (and the beautiful theory of Stallings automata for the free group 
case), but goes far beyond that: certain inductive procedures need a more general set- 
ting than mere subgroups, and rational subsets constitute the natural generalization. The 
connections between automata theory and group theory are rich and deep, and many are 
portrayed in Sims' book [53 1. 

This chapter is divided into three parts: in Section [T] we introduce basic concepts, 
terminology and notation for finitely generated groups, devoting special attention to free 
groups. These will also be used in Chapter l24l 

Section |2] describes the use of finite inverse automata to study finitely generated sub- 
groups of free groups. The automaton recognizes elements of a subgroup, represented as 
words in the ambient free group. 

Section [3] considers, more generally, rational subsets of groups, when good closure 
and decidability properties of these subsets are satisfied. 

The authors are grateful to Stuart Margolis, Benjamin Steinberg and Pascal Weil for 
their remarks on a preliminary version of this text. 



1 Finitely generated groups 

Let G be a group. Given A C G, let (A) = {A U A^''-)* denote the subgroup of G 
generated by A. We say that H ^ G is finitely generated and write H ^ /.g. G if 
H = {A) for some finite subset A of H. 

Given H ^ G,we denote by [G : H] the index of H in G, that is, the number of right 
cosets Hg for all g € G; or, equivalently, the number of left cosets. If [G : H] is finite, we 
write H ^/.i. G. It is well known that every finite index subgroup of a finitely generated 
group is finitely generated. 

We denote by 1 the identity of G. An element g G G has finite order if {g) is finite. 
Elements g,h G G are conjugate if h = x~^gx for some x € G. We use the notation 
g^ — h~^gh and [g, h] = g~^g'^ to denote, respectively, conjugates and commutators. 

Given an alphabet A, we denote by A^^ a set of formal inverses of A, and write A = 
A U A~-^. We say that A is an involutive alphabet. We extend : A A~^ : a n> a^^ 
to an involution on A* through 

(a^^)"^ = a, (uvy^^v^^u^^ {a e A, u,v e A*) . 

If G = (A), we have a canonical epimorphism p : A* G, mapping a^^ £ A to 
a^^ G G. We present next some classical decidability problems: 

Definition 1.1. Let G — (A) be a finitely generated group. 

word problem: is there an algorithm that, upon receiving as input a word u G A*, deter- 
mines whether or not p{u) = 1? 

conjugacy problem: is there an algorithm that, upon receiving as input words u,v e A*, 
determines whether or not p{u) and p{v) are conjugate in G? 
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membership problem for K. C2'^: is there for every X G /C an algorithm that, upon 
receiving as input a word u 6 A*, determines whether or not p{u) G XI 

generalized word problem: is the membership problem for the class of finitely gener- 
ated subgroups of G solvable? ^ 

order problem: is there an algorithm that, upon receiving as input a word u ^ A*, 
determines whether p{u) has finite or infinite order? 

isomorphism problem for a class Q of groups: is there an algorithm that, upon receiv- 
ing as input a description of groups G, H ^ Q, decides whether or not G = HI 
Typically, Q may be a subclass of finitely presented groups (given by their presen- 
tation), or automata groups (see Chapter |24]| given by automata. 

We can also require complexity bounds on the algorithms; more precisely, we may 
ask with which complexity bound an answer to the problem may be obtained, and also 
with which complexity bound a witness (a normal form for the word problem, an element 
conjugating p{u) to p{v) in case they are conjugate, an expression of u in the generators 
of X in the generalized word problem) may be constructed. 

1.1 Free groups 

We recall that an equivalence relation ~ on a semigroup 5* is a congruence if a ~ 6 
implies ac ^ he and ca ^ cb for all a,b,c ^ S. 

Definition 1.2. Given an alphabet A, let ~ denote the congruence on A* generated by 
the relation 

{{aa-\l)\aeA}. (1.1) 

The quotient Fa = A* j ^ is the /ree group on A. We denote \>y Q : A* ^ Fa the 
canonical morphism u M> [u]^. 

Free groups admit the following universal property: for every map f : A G, there 
is a unique group morphism Fa — s- G that extends /. 

Alternatively, we can view (II. lb as a confluent length-reducing rewriting system on 
A*, where each word w & A* can be transformed into a unique reduced word w with no 
factor of the form aa~^, see ||9l. As a consequence, the equivalence 

u^v ^ u = v (m, wGA*) 

solves the word problem for Fa- 

We shall use the notation Ra ^ A*. It is well known that Fa is isomorphic to Ra 
under the binary operation 

u-k V — UV {u, V e Ra) ■ 
We recall that the length \g\ of g e F^i is the length of the reduced form of g, also denoted 

by g- 

The letters of A provide a natural basis for Fa ■ they generate Fa and satisfy no non- 
trivial relations, that is, all reduced words on these generators represent distinct elements 
of Fa- a group is free if and only if it has a basis. 
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Throughout this chapter, we assume A to be a finite alphabet. It is well known that free 
groups Fa and Fb are isomorphic if and only if = ^B. This leads to the concept of 
rank of a free group F: the cardinality of a basis of F, denoted by rk F. It is common to 
use the notation F„ to denote a free group of rank n. 

We recall that a reduced word u is cyclically reduced if uu is also reduced. Any 
reduced word u E Ra admits a unique decomposition of the form u — vwv~^ with w 
cyclically reduced. A solution for the conjugacy problem follows easily from this: first 
reduce the words cyclically; then two cyclically reduced words in Ra are conjugate if and 
only if they are cyclic permutations of each other. On the other hand, the order problem 
admits a trivial solution: only the identity has finite order. Finally, the generalized word 
problem shall be discussed in the following section. 



2 Inverse automata and Stallings' construction 

The study of finitely generated subgroups of free groups entered a new era in the early 
eighties when Stallings made explicit and effective a construction Il54ll that can be traced 
back to the early part of the twentieth century in Schreier's coset graphs (see ll53l and il24.1l i 
and to Serre's work 1461 . Stallings' seminal paper was built over immersions of finite 
graphs, but the alternative approach using finite inverse automata became much more 
popular over the years; for more on their link, see 12611 . An extensive survey has been 
written by Kapovich and Miasnikov l20l . 

Stallings' construction for H ^ /.g. Fa consists in taking a finite set of generators for 
H in reduced form, building the so-called^ower automaton and then proceeding to make 
this automaton deterministic through the operation known as Stallings foldings . This is 
clearly a terminating procedure, but the key fact is that the construction is independent 
from both the given finite generating set and the chosen folding sequence. A short simple 
automata-theoretic proof of this claim will be given. The finite inverse automaton S{H) 
thus obtained is usually called the Stallings automaton of H. Over the years, Stallings au- 
tomata became the standard representation for finitely generated subgroups of free groups 
and are involved in many of the algorithmic results presently obtained. 

Several of these algorithms are implemented in computer software, see e.g. CRAG Q, 
or the packages AUTOMATA and FGA in GAP lil4J . 

2.1 Inverse automata 

An automaton A over an involutive alphabet A is involutive if, whenever (p, a, q) is an 
edge of A, so is (q, a^^,p). Therefore it suffices to depict just the positively labelled 
edges (having label in A) in their graphical representation. 

Definition 2.1. An involutive automaton is inverse if it is deterministic, trim and has a 
single final state. 

If the latter happens to be the initial state, it is called the basepoint. It follows easily 
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from the computation of the Nerode equivalence (see §10.2) that every inverse automaton 
is a minimal automaton. 

Finite inverse automata capture the idea of an action (of a finite inverse monoid, their 
transition monoid) on a finite set (the vertex set) through partial bijections. We recall 
that a monoid M is inverse if, for every x G M, there exists a unique y € M such that 
xyx = X and y = yxy; then M acts by partial bijections on itself. 
The next result is easily proven, but is quite useful. 

Proposition 2.1. Let A be an inverse automaton and let p-!^ — be a path in A. 
Then there exists also a path p-^q in A. 

Another important property relates languages to morphisms. For us, a morphism be- 
tween deterministic automata A and is a mapping 93 between their respective vertex 
sets which preserves initial vertices, final vertices and edges, in the sense that (<^(p) , a, (/'(s)) 
is an edge of A! whenever (p, a, q) is an edge of A. 

Proposition 2.2. Given inverse automata A and A', then L(A) C L(A') if and only if 
there exists a morphism ip : A^ A'. Moreover, such a morphism is unique. 

Proof. Given a vertex q of A, take a successful path 

qo — >q — >t 

in A, for some u,v £ A*. Since L{A) C L{A'), there exists a successful path 

5o — >1 — 

in A'. We take tp{q) = q'. 

To show that (fi is well defined, suppose that 

u' v' , 

qo — >q — H 

is an alternative successful path in A. Since u'v G L{A) C L{A'), there exists a success- 
ful path 

^q'o^q"^t'^ 

in A' and it follows that q' = q" since A' is inverse. Thus (fi is well defined. 

It is now routine to check that cpis a morphism from A to A' and that it is unique. 
('^=): Immediate from the definition of morphism. □ 



2.2 Stallings' construction 

Let X be a finite subset of Ra- We build an involutive automaton J^{X) by fixing a 
basepoint qo and gluing to it a petal labelled by every word in X as follows: if a; = 
ai . . .ak & X, with G A, the petal consists of a closed path of the form 

01, 0,2, ak, 

qo — > • — > yqo 

and the respective inverse edges. AH such intermediate vertices • are assumed to be 
distinct in the automaton. For obvious reasons, J^{X) is called the flower automaton of 
X. 
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The automaton J-{X) is almost an inverse automaton - except that it need not be 
deterministic. We can fix it by performing a sequence of so-called Stallings foldings. 
Assume that ^ is a trim involutive automaton with a basepoint, possessing two distinct 
edges of the form 

P — P — >r (2.1) 

for a E A. The folding is performed by identifying these two edges, as well as the two 
respective inverse edges. In particular, the vertices q and r are also identified (if they were 
distinct). 

The number of edges is certain to decrease through foldings. Therefore, if we perform 
enough of them, we are sure to turn J-{X) into a finite inverse automaton. 

Definition 2.2. The Stallings automaton of X is the finite inverse automaton S{X) ob- 
tained through folding J-{X). 

We shall see that S{X) depends only on the finitely generated subgroup {X) of Fa 
generated by X, being in particular independent from the choice of foldings taken to reach 
it. 

Since inverse automata are minimal, it suffices to characterize L(S{X)) in terms of 
H to prove uniqueness (up to isomorphism): 

Proposition 2.3. Fix H ^ y Fa and let X C Ra be a finite generating set for H. Then 

L{S{X)) = (^{i C A* I L is recognized by a finite inverse automaton 

with a basepoint and H C_ L} . 

Proof. (3): Clearly, S{X) is a finite inverse automaton with a basepoint. Since X U 
C L{F{X)) C it follows easily from PropositionOthat 

H C L{S{X)) . (2.2) 

(C): Let L C A* be recognized by a finite inverse automaton A with a basepoint, 
with H C L. Since X C H, we have an automaton morphism from F{X) to A, hence 
L{F{X)) C L. To prove that L{S{X)) C L, it suffices to show that inclusion in L is 
preserved through foldings. 

Indeed, assume that L{B) C L and B' is obtained from B by folding the two edges 
in ( 12.1b . It is immediate that every successful path q^-^t in B' can be lifted to a success- 
ful path qo-^t in B by successively inserting the word a~^a into u. Now v E L = L{A) 
implies u £ L in view of Proposition l2.1l □ 



Now, given H ^ Fa finitely generated, we take a finite set X of generators. Without 
loss of generality, we may assume that X consists of reduced words, and we may define 
S{H) — S{X) to be the Stallings automaton of H. 

Example 2.1. Stallings' construction for X = {a^^ba, ba^}, where the next edges to be 
identified are depicted by dotted lines, is 
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A simple, yet important example is given by applying the construction to F„ itself, 
when we obtain the so-called bouquet of n circles: 



c 




S{F2) S{F3) 

la terms of complexity, the best known algorithm for the construction of S{X) is due 
to Touikan ll56l . Its time complexity is 0{n log* n), where n is the sum of the lengths of 
the elements of X. 



2.3 Basic applications 

The most fundamental application of Stallings' construction is an elegant and efficient 
solution to the generalized word problem: 

Theorem 2.4. The generalized word problem in Fa is solvable. 

We will see many groups in Chapter |24] that have solvable word problem; however, 
few of them have solvable generalized word problem. The proof of Theorem l2.4l relies on 

Proposition 2.5. Consider H g Fa and u G Fa- Then u G H if and only ifuE 
L{S{H)). 

Proof. (^): Follows from i2.2i . 

(<J=): It follows easily from the last paragraph of the proof of Proposition 12 . 3 1 that, if 
B' is obtained from B by performing Stallings foldings, then L{B') — L{B). Hence, if 
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H = (X), we get 

L{S{H)) = L{I-{X)) = (X U X-i)* = H 
and the implication follows. □ 

It follows from our previous remark that the complexity of the generalized word prob- 
lem is 0(nlog* n + m), where n is the sum of the lengths of the elements of X and 
m is the length of the input word. In particular, once the subgroup X has been fixed, 
complexity is linear in m. 

Example 2.2. We may use the Stallings automaton constructed in Example 12. II to check 
that baba^^b^^ e H ^ (a"^6a, ba'^) but ab ^ H. 

Stallings automata also provide an effective construction for bases of finitely generated 
subgroups. Consider H ^f.g. Fa, and let m be the number of vertices of S{H). A 
spanning tree T for S{H) consists of m — 1 edges and their inverses which, together, 
connect all the vertices of S{H). Given a vertex p of S{H), we denote by gp the T- 
geodesic connecting the basepoint go to p, that is, qo—^p is the shortest path contained 
in T connecting go to p. 

Proposition 2.6. Let H ^ f ,g. Fa and let T be a spanning tree for S{H). Let be the 
set of positively labelled edges ofS{H). Then H is free with basis 

y = {gpO-Qq^ I {p, a, q) e E+\T}. 

Proof. It follows from Proposition 12 . 5 1 that L{S{H)) C H, hence Y C H. To show that 
H = (Y), take h — ai ■ ■ ■ at G -ff in reduced form (a^ G A). By Proposition |23] there 
exists a successful path 

qo — >qi — > >qk = qo 

in S{H). For i — 1, . . . , fc, we have either gqi_iaig^/^ £ Y L) Y^^ or gqi_iaigqi^ — 1, 
the latter occurring if {qi-i, Ui, qi) e T. In any case, we get 

/l = ai • • • flfe = igqoai9qi^)i9qia2gq2^) ■ ■ ■ igqk-iO-kgqo) £ {Y) 

and soTJ = {Y). 

It remains to show that the elements of Y satisfy no nontrivial relations. Let yi, . . . ,yk 
£Y Li Y~^ with yi ^ for i = 2, . . . , fc. Write yi — gp.aig~^, where ai £ A labels 
the edge not in T. It follows easily from yi ^ and the definition of spanning tree 

that 

yi • • • 2/fe = gpi_aigrHvia2 ■ ■ ■ ak-igrk\i9pkakgrk , 
a nonempty reduced word if fc ^ 1. Therefore F is a basis of H as claimed. □ 

In the process, we also obtain a proof of the Nielsen-Schreier Theorem, in the case of 
finitely generated subgroups. A simple topological proof may be found in l36l : 

Theorem 2.7 (Nielsen-Schreier). Every subgroup of a free group is itself free. 
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Example 2.3. We use the Stallings automaton constructed in Example 12. II to construct a 

basis of H = {a^^ba, ba^). 

If we take the spanning tree T defined by the dotted lines in 

/ — ■ 

© . o 

then \ T = 2 and the corresponding basis is {ba^, baba^^b^^}. Another choice of 
spanning tree actually proves that the original generating set is also a basis. 

We remark that Proposition 12.61 can be extended to the case of infinitely generated 
subgroups, proving the general case of Theorem l2.7l However, in this case there is no ef- 
fective construction such as Stallings', and the (infinite) inverse automaton S{H) remains 
a theoretical object, using appropriate cosets as vertices. 

Another classical application of Stallings' construction regards the identification of 
finite index subgroups. 

Proposition 2.8. Consider H ^ /.g. Fa- 

(i) H is a finite index subgroup of Fa if and only if S{H) is a complete automaton. 

(ii) If H is a finite index subgroup of Fa, then its index is the number of vertices of 
S{H). 

Proof, (i) (=>): Suppose that S{H) is not complete. Then there exist some vertex q and 
some a G j4 such that g ■ a is undefined. Let g be a geodesic connecting the basepoint go 
to q in S{H). We claim that 

HgaJ^ ^ i?.ga" if m - n > \g\ . (2.3) 

Indeed, HgaJ^ = Hga^ impHes .ga™-"g-i G H and so ga'^-^g^^ G L{S{H)) by 
Proposition l2.5l Since ga is reduced due to S{H) being inverse, it follows from m — n> 
\g\ that gaa'"^^^^^g^^ = ga''"-^^g^^ G L{S{H)): indeed, g^^ is not long enough to 
erase all the a's. Since S{H) is deterministic, q ■ a must be defined, a contradiction. 
Therefore (I2.3l l holds and so H has infinite index. 

(<S=): Let Q be the vertex set of S{H) and fix a geodesic qo~^q for each q ^ Q. 
Take u G Fa- Since S{H) is complete, we have a path qo—^q for some q ^ Q. Hence 
ug~^ G H and so w = ug~^gq G Hgq. Therefore Fa = U^eg ^.9<j so H ^ f,i. Fa- 

(ii) In view of Fa = Uggg Hgq, it suffices to show that the cosets Hgq are all distinct. 
Indeed, assume that Hgp ~ Hgq for some vertices p,q E Q. Then gpgq^ G H and so 

9p9q^ £ L{S{H)) by Proposition 12.51 On the other hand, since S{H) is complete, we 
have a path _ , 

qo >r 
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for some r G Q. In view of Proposition 12. II and by determinism, we get r — qq. Hence 
we have paths _i _i 

p >qo, q >qQ ■ 

Since S{H) is inverse, we get p = q as required. □ 

Example 2.4. Since the Stallings automaton constructed in Example l2.1l is not complete, 
it follows that (a^^ba, ha?) is not a finite index subgroup of F2. 

Corollary 2.9. If H ^ Fa has index n, then ikH =1 + n{=f^A - 1). 

Proof. By Proposition l2.8l the automaton S{H) has n vertices and n^A positive edges. 
A spanning tree has n — 1 positive edges, so rk_ff = n^A — (n — 1) = 1 + n{^A — 1) 
by Proposition l2.6l □ 

Beautiful connections between finite index subgroups and certain classes of bifix codes 
— set of words none of which is a prefix or a suffix of another — have recently been 
unveiled by Berstel, De Felice, Perrin, Reutenauer and Rindone |61. 



2.4 Conjugacy 

We start now a brief discussion of conjugacy. Recall that the outdegree of a vertex q is the 
number of edges starting at q and the geodesic distance in a connected graph is the length 
of the shortest undirected path connecting two vertices. 

Since the original generating set is always taken in reduced form, it follows easily that 
there is at most one vertex in a Stallings automaton having outdegree < 2: the basepoint 
go- Assuming that H is nontrivial, S{H) must always be of the form 




where qi is the closest vertex to go (in terms of geodesic distance) having outdegree > 2 
(since there is at least one vertex having such outdegree). Note that qi — qo if qo has 
outdegree > 2 itself. We call qo-^ the tail (which is empty if qi = qo) and the remaining 
subgraph the core of S{H). 

Note that S{H), and its core, may be understood as follows. Consider the graph with 
vertex set Fa/H — {gH \ g e Fa}, with an edge from gH to agH for each generator 
a € A. Then this graph, called the Schreier graph (see i]24.1b of H\Fa, consists of 
finitely many trees attached to the core of S{H). 

Theorem 2.10. There is an algorithm that decides whether or not two finitely generated 
subgroups of Fa are conjugate. 
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Proof. Finitely generated subgroups G, H are conjugate if and only if the cores of S{G) 
and S{H) are equal (up to their basepoints). □ 

The Stallings automata of the conjugates of H can be obtained in the following ways: 
(1) declaring a vertex in the core C to be the basepoint; (2) gluing a tail to some vertex in 
the core C and taking its other endpoint to be the basepoint. 

Note that the tail must be glued in some way that keeps the automaton inverse, so 
in particular this second type of operation can only be performed if the automaton is 
not complete, or equivalently, if H has infinite index. An immediate consequence is the 
following classical 

Proposition 2.11. A finite rank normal subgroup of a free group is either trivial or has 
finite index. 

Moreover, a finite index subgroup H is normal if and only if its Stallings automaton 
is vertex-transitive, that is, if all choices of basepoint yield the same automaton. 

Example 2.5. StalUngs automata of some conjugates of H = {a^^ba, ba^): 




We can also use the previous discussion on the structure of (finite) Stalhngs automata to 
provide them with an abstract characterization. 

Proposition 2.12. A finite inverse automaton with a basepoint is a Stallings automaton if 
and only if it has at most one vertex of outdegree 1: the basepoint. 



Proof. Indeed, for any such automaton we can take a spanning tree and use it to construct 
a basis for the subgroup as in the proof of Proposition l2.6l □ 
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2.5 Further algebraic properties 

The study of intersections of finitely generated subgroups of Fa provides further applica- 
tions of Stallings automata. Howson's classical theorem admits a simple proof using the 
direct product of two Stallings automata; it is also an immediate consequence of Theorem 
Oand CorollaryElii). 

Theorem 2.13 (Howson). IfH, K ^ /.g. Fa, then also HnK ^ f,g. Fa- 

Stallings automata are also naturally related to the famous Hanna Neumann conjec- 
ture: given H,K ^ f,g. Fa, then rk(i? n X) - 1 (rkiJ - \){y\K - 1). The conjec- 
ture arose in a paper of Hanna Neumann |34|, where the inequality rk(i7 n K) — 1 ^ 
2(rki? — l){ikK — 1) was also proved. In one of the early applications of Stallings' 
approach, Gersten provided an alternative geometric proof of Hanna Neumann's inequal- 
ity lil5]. 

A free factor of a free group Fa can be defined as a subgroup H generated by a subset 
of a basis of Fa- This is equivalent to saying that there exists a/ree product decomposition 
Fa=H*K for some K Fa- 

Since the rank of a free factor never exceeds the rank of the ambient free group, it is 
easy to construct examples of subgroups which are not free factors: it follows easily from 
Proposition l2.6l that any free group of rank ^ 2 can have subgroups of arbitrary finite rank 
(and even infinite countable). 

The problem of identifying free factors has a simple solution based on Stallings au- 
tomata 150|: one must check whether or not a prescribed number of vertex identifications 
in the Stallings automaton can lead to a bouquet. However, the most efficient solution, due 
to Roig, Ventura and Weil |40|, involves Whitehead automorphisms and will therefore be 
postponed to i)23 l2.7l 

Given a morphism ip : A ^ B of inverse automata, let the morphic image f{A) be 
the subautomaton of B induced by the image by tp of all the successful paths of A. 

The following classical result characterizes the extensions of H ^ f,g. Fa contained 
in Fa ■ We present the proof from ll32l : 

Theorem 2.14 (Takahasi [55]). Given H Sj/.g. Fa, one can effectively compute finitely 
many extensions Ki , . . . , Km ^ f.g. Fa of H such that the following conditions are equiv- 
alent for every K ^ /.g. Fa: 

(i) H < K; 

(ii) Ki is a free factor of K for some i G {1, . . . , m}. 

Proof. Let Ai, . . . ,Am denote all the morphic images of S{H), up to isomorphism. 
Since a morphic image cannot have more vertices than the original automaton, there are 
only finitely many isomorphism classes. Moreover, it follows from Proposition l2. 12l that. 
for i = 1, . . . ,TO, A = S{K,) for some K, </.g. Fa- Since L{S{H)) C L{At) = 
L{S{Ki)), it follows from Proposition l2.5l that H ^ Ki. Clearly, we can construct all Ai 
and therefore all Ki. 

(i) =^ (ii). If H K, it follows from StalUngs' construction that L{S{H)) C 
L(S{K)) and so there is a morphism p : S{H) — > S{K) by Proposition 12.21 Let Ai 
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be, up to isomorphism, the morphic image of S{H) through if. Since Ai — S{Ki) is a 
subautomaton of S{K), it follows easily from Proposition 12.61 that Ki is a free factor of 
K: it suffices to take a spanning tree for S{Ki), extend it to a spanning tree for S{K), 
and the induced basis of Ki will be contained in the induced basis of K. 



An interesting research line related to this result is built on the concept of algebraic 
extension, introduced by Kapovich and Miasnikov [i20|, and inspired by the homonymous 
field-theoretical classical notion. Given H ^ K ^ Fa, we say that K is an algebraic 
extension of H if no proper free factor of K contains H. Miasnikov, Ventura and Weil 
II32I proved that the set of algebraic extensions of H is finite and effectively computable, 
and it constitutes the minimum set of extensions Ki, . . . , Km satisfying the conditions of 
Theorem irT4l 

Consider a subgroup H of a group G. The commensurator of H in G, is 

CommG(i?) = {5 e G I i? n iJ^* has finite index in H and H^. (2.4) 

For example, the commensurator of GL„(Z) in GL„(IR) is GL„(Q). 

The special case of finite-index extensions, H ^ / , K ^ Fa is of special interest, 
and can be interpreted in terms of commensurators. It can be proved (see |2()', Lemma 
8.7] and |52|) that every H ^ / g Fa has a maximum finite-index extension inside Fa, 
denoted by Hfi, and Hfi — Commi?^ (iJ). Silva and Weil ll52l proved that S{Hfi) can 
be constructed from S{H) using a simple automata-theoretic algorithm: 

(1) The standard minimization algorithm is applied to the core of S(H), taking all 
vertices as final. 

(2) The original tail of S{H) is subsequently reinstated in this new automaton, at the 
appropriate vertex. 

We present now an application of different type, involving transition monoids. It 
follows easily from the definitions that the transition monoid of a finite inverse automaton 
is always a finite inverse monoid. Given a group G, we say that a subgroup H ^ G is 
pure if the implication 



holds for all g G Fa and n ^ 1. If p is a prime, we say that H is p-pure if ( 12.5b holds 
when {n,p) = 1. 

The next result is due to Birget, Margolis, Meakin and Weil, and is the only natural 
problem among applications of Stallings automata that is known so far to be PSPACE- 
complete ID. 

Proposition 2.15. For every H ^ j.g. Fa, the following conditions hold: 

(i) H is pure if and only if the transition monoid of S{H) is aperiodic. 

(ii) H is p-pure if and only if the transition monoid of S{H) has no subgroups of order 
P- 



(ii) (i) is immediate. 



□ 



(2.5) 



Proof. Both conditions in (i) are easily proved to be equivalent to the nonexistence in 
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S{H) of a cycle of the form 

u 

(T^^^^^^^J^ (fc^l,p^g) 
where u can be assumed to be cyclically reduced. The proof of (ii) runs similarly. □ 



2.6 Topological properties 

We require for this subsection some basic topological concepts, which the reader can 
recover from Chapter 17. 

For all u, w g Fa, written in reduced form as elements of Ra, let u A u denote the 
longest common prefix of u and v. The prefix metric d on Fa is defined, for all u,v ^ Fa, 
by 

2-\uAv\-l Ifu^v 

if 7i = u 



d{u, v) 



It follows easily from the definition that d is an ultrametric on Fa, satisfying in particular 
the axiom 

d{u, v) ^ max{(i(u, w), d{w, v)} . 

The completion of this metric space is compact; its extra elements are infinite reduced 
words aia2a3 . . . , with all G A, and constitute the hyperbolic boundary OFa of Fa, 
see i]24.1.5l Extending the operator A to Fa U OFa in the obvious way, it follows easily 
from the definitions that, for every infinite reduced word a and every sequence (m„)„ in 
Fa, 

a= lim Un if and only if lim |q!Au„| = +oo. (2.6) 

The next result shows that Stallings automata are given a new role in connection with 
the prefix metric. We denote by cl H the closure of H in the completion of Fa- 

Proposition 2.16. IfH ^ /.^ Fa, then cl H is the union ofH with the set of all a G OFa 

that label paths in S{H) out of the basepoint. 



Proof. Since the topology of Fa is discrete, we have c\H H Fa = H. 

(C): If a E OFa does not label a path in S{H) out of the basepoint, then {\a A h\ : 
h G iJ} is finite and so no sequence of H can converge to a by (12.6b . 

(3): Let a — aia2a3 • • • G OFa, with G A, label a path in S{H) out of the 
basepoint. Let m be the number of vertices of S{H). For every n ^ 1, there exists some 
word Wn of length < m such that oi • • • a„K7„ G H. Now a — lim„_>.+oo ai • • • a„w„ 
by and so a G cli?. □ 

The profinite topology on Fa is defined in Chapter 17: for every u G Fa, the collection 
{Ku I K ^f.i. Fa} constitutes a basis of clopen neighbourhoods of u. In his seminal 
1983 paper Ii54i . Stallings gave an alternative proof of Marshall Hall's Theorem; 
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Theorem 2.17 (M. Hall). Every finitely generated subgroup of Fa is closed for the profi- 
nite topology. 

Proof. Fix H ^ /.g. Fa and let u € Fa \ H he written in reduced form as an element 
of Ra- In view of Proposition 12.51 u does not label a loop at the basepoint qo of S{H). 
If there is no path qq — > ■ • ■ in S{H), we add new edges to S{H) to get a finite inverse 
automaton A having a path qo~-^q qo- Otherwise just take A = S{H). Next add 
new edges to A to get a finite complete inverse automaton B. In view of Propositions 12. 81 
and 12.121 we have B ~ S{K) for some K ^ /.i. Fa- Hence Ku is open and contains u. 
Since H n Ku ^ yields u G K^^H = K, contradicting Proposition l2.5l it follows that 
H n Ku ~ and so H is closed as claimed. □ 



Example 2.6. We consider the above construction for H = {a ^ba, ba^) and u = b^: 




b 



If we take the spanning tree defined by the dotted lines in B, it follows from Proposi- 
tion |2]6] that 

K = {ba-\b^,b^ab-^, ba^ , baba'^b-^) 
is a finite index subgroup of F2 such that H n Kb^ = 0. 

We recall that a group G is residually finite if its finite index subgroups have trivial 
intersection. Considering the trivial subgroup in Theorem l2.17l we deduce 

Corollary 2.18. Fa is residually finite. 

We remark that Ribes and Zalessky extended Theorem 12.171 to products of finitely 
many finitely generated subgroups of Fa, see [38 1. This result is deeply connected to the 
solution of Rhodes' Type II conjecture, see ifJTl Chapter 4]. 

If V denotes a pseudovariety of finite groups (see Chapter 16), the pro-Y topology on 
Fa is defined by considering that each u e Fa has 

{Ku\K^f,,,FA, FA/KeY} 
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as a basis of clopen neighbourhoods. The closure for the pro-V topology of H ^ f,g Fa 
can be related to an extension property of S{H), and Margolis, Sapir and Weil used 
automata to prove that efficient computation can be achieved for the pseudovarieties of 
finite p-groups and finite nilpotent groups Il28ll . The original computability proof for the 
p-group case is due to Ribes and Zalessky ||39l . 

2.7 Dynamical properties 

We shall mention briefly some examples of applications of Stallings automata to the study 
of endomorphism dynamics, starting with Gersten's solution of the subgroup orbit prob- 
lem [16|. 

The subgroup orbit problem consists in finding an algorithm to decide, for given H, K 
^ f.g. Fa, whether or not K = f{H) for some automorphism (f of Fa- Equivalently, this 
can be described as deciding whether or not the automorphic orbit of a finitely generated 
subgroup is recursive. 

Gersten's solution adapts to the context of Stallings automata Whitehead's idea to 
solve the orbit problem for words |59|. Whitehead's proof relies on a suitable decom- 
position of automorphisms as products of elementary factors (which became known as 
Whitehead automorphisms), and on using these as a tool to compute the elements of min- 
imum length in the automorphic orbit of the word. In the subgroup case, word length is 
replaced by the number of vertices of the Stallings automaton. 

The most efficient solution to the problem of identifying free factors f40l, mentioned 
in §23|23] also relies on this approach: H ^/.g. Fa is a free factor if and only if the 
Stallings automaton of some automorphic image of H has a single vertex (that is, a bou- 
quet). 

Another very nice application is given by the following theorem of Goldstein and 
Turner lini: 

Theorem 2.19. The fixed point subgroup of an endomorphism of Fa is finitely generated. 

Proof. Let ip be an endomorphism of Fa- For every u e Fa, define Q{u) — (p{u)u^^. 
We define a potentially infinite automaton A by taking 

{Q{u) I ueFAjC Fa 

as the vertex set, all edges of the form Q{u)-^Q{au) with u G Fa, a ^ A, and fixing 1 
as the basepoint. Then ^ is a well-defined inverse automaton. 

Next we take B to be the subautomaton of A obtained by retaining only those vertices 
and edges that lie in successful paths labelled by reduced words. Clearly, B is still an 
inverse automaton, and it is easy to check that it must be the Stallings automaton of the 
fixed point subgroup of (p. 

It remains to be proved that B is finite. We define a subautomaton C of ;B by removing 
exactly one edge among each inverse pair 

Q{u)'^Q{au), Q{au)— — >Q{u) 

with a G A as follows: if a^^ i^s the last letter of Q{au), we remove Q{u)-^Q{au); 
otherwise, we remove Q{au)— — >Q{u). 
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Let M denote the maximum length of the image of a letter by Lp. We claim that, 
whenever > 27\^, the vertex Q{v) has outdegree at most 1. 

Indeed, if Q{v)~ — )-Q{a~^v) is an edge in C for a ^ A, then a^^ is the last letter of 
Q{v). On the other hand, if Q{v) — >Q{hv) is an edge in C for b ^ A, then b^^ is not the 
last letter of (5(6i;). Since. Q{bv) = if{b)Q{v)b^^ and \Q{v)\ > 2|(/3(6)|, then 6 must be 
the last letter of Q{v) in this case. Since Q{v) has at most one last letter, it follows that 
its outdegree is at most 1 . 

Let D be a finite subautomaton of C containing all vertices Q{v) such that ^ 
2M. Suppose that p — ¥q is an edge in C not belonging to V. Since p — >q, being an edge 
of B, must lie in some reduced path, and by the outdegree property of C, it is easy to see 
that there exists some path in C of the form 

p' — >p — >q — >r< — r' 

where p', r' are vertices in V. Since there are only finitely many directed paths out of V, 
it follows that C is finite and so is B. Therefore the fixed point subgroup of (p is finitely 
generated. □ 

Note that this proof is not by any means constructive. Indeed, the only known al- 
gorithm for computing the fixed point subgroup of a free group automorphism is due to 
Maslakova ||3TI and rehes on the sophisticated train track theory of Bestvina and Han- 
del |7| and other algebraic geometry tools. The general endomorphism case remains 
open. 

Stallings automata were also used by Ventura in the study of various properties of 
fixed subgroups, considering in particular arbitrary families of endomorphisms Il57l [30l 
(see also |58]). 

Automata also play a part in the study of infinite fixed points, taken over the continuous 
extension of a monomorphism to the hyperbolic boundary (see for example P9l ). 



3 Rational and recognizable subsets 

Rational subsets generalize the notion of finitely generated from subgroups to arbitrary 
subsets of a group, and can be quite useful in establishing inductive procedures that need 
to go beyond the territory of subgroups. Similarly, recognizable subsets extend the notion 
of finite index subgroups. Basic properties and results can be found in ||5l or ll43ll . 

We consider a finitely generated group G = (A), with the canonical map tt : Fa G. 
A subset of G is rational if it is the image by p = 7r0 of a rational subset of A*, and is 
recognizable if its full preimage under p is rational in A* . 

For every group G, the classes Rat G and Rec G satisfy the following closure proper- 
ties: 

• Rat G is (effectively) closed under union, product, star, morphisms, inversion, sub- 
group generating. 

• Rec G is (effectively) closed under boolean operations, translation, product, star, 
inverse morphisms, inversion, subgroup generating. 
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Kleene's Theorem is not valid for groups: Rat G — Rec G if and only if G is finite. 
However, if the class of rational subsets of G possesses some extra algorithmic properties, 
then many decidability/constructibility results can be deduced for G. Two properties are 
particularly coveted for Rat G: 

• (effective) closure under complement (yielding closure under all the boolean oper- 
ations); 

• decidable membership problem for arbitrary rational subsets. 

In these cases, one may often solve problems (e.g. equations, or systems of equations) 
whose statement lies far out of the rational universe, by proving that the solution is a 
rational set. 



3.1 Rational and recognizable subgroups 

We start by some basic, general facts. The following result is essential to connect language 
theory to group theory. 

Theorem 3.1 (Anisimov and Seifert). A subgroup Hofa group G is rational if and only 
if H is finitely generated. 

Proof. Let iJ be a rational subgroup of G and let vr : Fa ~> G denote a morphism. 
Then there exists a finite A-automaton A such that H = p{L{A)). Assume that A has m 
vertices and let X consist of all the words in p'~^{H) of length < 2m. Since A is finite, 
so is X. We claim that H — {p{X)). To prove it, it suffices to show that 

u e L{A) p{u) e {p{X)) (3.1) 

holds for every u ^ A*. We use induction on By definition of X, (13.1b holds for 
words of length < 2m. Assume now that |u| ^ 2m and (13.1b holds for shorter words. 
Write u = vw with \w\ = m. Then there exists a path 

V z , 

qa — >q — >t 

in A with \z\ < m. Thus vz e L{A) and by the induction hypothesis p{vz) G {p{X)). 
On the other hand, |z~^u)| < 2m and p{z^^w) — p{z^^v^^)p{vw) e H, hence z~^w G 
X and so p{u) — p{vz)p{z^^w) G {p{X)), proving ( 13.1b as required. 

(<^) is trivial. □ 

It is an easier task to characterize recognizable subgroups: 

Proposition 3.2. A subgroup H of a group G is recognizable if and only if it has finite 
index. 

Proof. (^): In general, a recognizable subset of G is of the form NX, where N < / i. G 
and X C_ G '\s finite. If H = NX is a subgroup of G, then N C H and so H has finite 
index as well. 

(<^): This follows from the well-known fact that every finite index subgroup H of G 
contains a finite index normal subgroup N of G, namely N ~ PlggG 9^9^- Since N 
has finite index, H must be of the form NX for some finite X G. □ 
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3.2 Benois' Theorem 

The central result in this subsection is Benois' Theorem, the cornerstone of the whole 
theory of rational subsets of free groups: 

Theorem 3.3 (Benois). 

(i) If L C A* is rational, then L is also rational, and can be effectively constructed 
from L. ^ 

(ii) A subset of Ra is a rational language as a subset of A* if and only if it is rational 
as a subset of Fa- 

We illustrate this in the case of finitely generated subgroups: temporarily calling 
"Benois automata" those automata recognizing rational subsets of Ra, we may convert 
them to Stallings automata by "folding" them, at the same time making sure they are in- 
verse automata. Given a Stallings automaton, one intersects it with Ra to obtain a Benois 
automaton. 

Proof, (i) Let A = {Q, A, E, I, T) be a finite automaton recognizing L. We define a 
sequence {An)n of finite automata with e-transitions as follows. Let ~ A. Assuming 
that An = {Q,A, En, I, T) is defined, we consider all instances of ordered pairs (p, q) e 
Q X Q such that 

there exists a path p— — >q in An for some a £ A, but no path p—^q. (P) 

Clearly, there are only finitely many instances of (P) in An- We define En+i to be the 
union of £"„ with all the new edges {p, 1, q), where {p, q) € Q x Q is an instance of (P). 
Finally, we define An+i = {Q, A, En+i,I, T). In particular, note that An — An+k for 
every /c ^ 1 if there are no instances of (P) in An- 

Since Q is finite, the sequence {An)n is ultimately constant, say after reaching Am- 
We claim that 

L^L{Anr)r\RA- (3.2) 

Indeed, take u E L- There exists a sequence of words u = uq, mi, . . . , Uk-i, Uk — u 
where each term is obtained from the preceding one by erasing a factor of the form aa~^ 
for some a G A. A straightforw^a|rd induction shows that Ui G L{Ai) for i = 0, . . . , fc, 
since the existence of a path p^ — >q in Ai implies the existence of a path p — >q in Ai+i- 
Hence u = e L{Ak) C L{Am) and it follows that L C L{A,n) H Ra- 

For the opposite inclusion, we start by noting that any path p-^q in Ai+i can be 
hfted to a path p-^q in Ai, where v is obtained from u by inserting finitely many factors 
of the form aa~^- It follows that 

L(A«) - L{A,n-i) L{Aq) ^ L 

and so L{A„i) n i?A C L{Am) = L- Thus ([iJll holds. 
Since _ _ _ 

RA^A*\[j A*aa-^A* 

is obviously rational, and the class of rational languages is closed under intersection, it 
follows that L is rational. Moreover, we can effectively compute the automaton Am and 
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a finite automaton recognizing Ra, hence the direct product construction can be used to 
construct a finite automaton recognizing the intersection L = L{Am) H Ra- 

(ii) Consider X C Ra. If X G Rat A*, then 9{X) G Rat Fa and so X is rational as 
a subset of Fa- 

Conversely, if X is rational as a subset of Fa, then X = 0{L) for some L G Rat A*. 
Since X C i?^, we get X = L. Now part (i) yields L G Rat A* and so X G Rat A* as 
required. □ 



Example 3.1. Let ^ = be depicted by 




and we can then proceed to compute L = L{A2) H R2- 

The following result summarizes some of the most direct consequences of Benois' 
Theorem: 

Corollary 3.4. 

(i) Fa has decidable rational subset membership problem. 

(ii) Rat Fa is closed under the boolean operations. 

Proof, (i) Given X G Rati^^ and u G Fa, write X = 9{L) for some L G Rat A*. 
Then u E X \f and only ifu^X — L. By Theorem 13. Sf i). we may construct a finite 
automaton recognizing L and therefore decide whether or not u ^ L. 

(ii) Given X G Rat Fa, we have Fa\X = Ra\X and so Fa \ X G Rat Fa by 
Theorem l3.3l Therefore Rat Fa is closed under complement. 

Since Rat Fa is trivially closed under union, it follows from De Morgan's laws that it 
is closed under intersection as well. □ 

Note that we can associate algorithms to these boolean closure properties of Rat Fa in 
a constructive way. We remark also that the proof of Theorem l3.3l can be clearly adapted to 
more general classes of rewriting systems (see [9J). Theorem l3.3l and Corollary [33] have 



Rational subsets of groups 



21 



been generalized several times by Benois herself lID and by Senizergues, who obtained 
the most general versions. Senizergues' results ll44l hold for rational length-reducing 
left basic confluent rewriting systems and remain valid for the more general notion of 
controlled rewriting system. 

3.3 Rational versus recognizable 

Since Fa is a finitely generated monoid, it follows that every recognizable subset of 
Fa is rational |5 Proposition III.2.4]. We turn to the problem of deciding which ra- 
tional subsets of Fa are recognizable. The first proof, using rewriting systems, is due 
to Senizergues [45 .1 but we follow the shorter alternative proof from [48 il . where a third 
alternative proof, of a more combinatorial nature, was also given. 

Given a subset X of a group G, we define the right stabilizer of X to be the submonoid 
of G defined by 

R{X) = {g e G I Xg C X} . 

Next let 

K{X) = R{X) n (R{X))-^ = {.g e G I Xg = X} 
be the largest subgroup of G contained in R{X) and let 

N{X) = fl 9K{X)g-^ 
aeG 

be the largest normal subgroup of G contained in K{X), and therefore in R{X). 

Lemma 3.5. A subset X of a group G is recognizable if and only if K{X) is a finite index 
subgroup ofG. 

In fact, the Schreier graph (see i ]24.1l ) of K{X)\G is the underlying graph of an 
automaton recognizing X, and G/N{X) is the syntactic monoid of X. 

Proof (^): If X C G is recognizable, then X ^ NF for some N < G and F C G 
finite. Hence N C R{X) and so iV C K{X) since N ^G. Since N has finite index in 
G, so does K{X). 

{<=): If K{X) is a finite index subgroup of G, so is = N{X). Indeed, a finite 
index subgroup has only finitely many conjugates (having also finite index) and a finite 
intersection of finite index subgroups is easily checked to have finite index itself. 

Therefore it suffices to show that X — FN for some finite subset F of G. Since N 
has finite index, the claim follows from XN = X, in turn an immediate consequence of 
N C R{X). □ 

Proposition 3.6. It is decidable whether or not a rational subset of Fa is recognizable. 

Proof. Take X G Rat Fa- In view of Lemma l3.5l and Proposition |23] it suffices to show 
that K{X) is finitely generated and effectively computable. 
Given u G Fa, we have 

u i R[X)^Xu (IX ^ Xu<r\{FA\X) + %^u^ X-\Fa\X), 
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hence 

R{X) = Fa\{X-\Fa\X)). 

It follows easily from the fact that the class of rational languages is closed under reversal 
and morphisms, combined with Theorem l3.31 ii). that X^^ G Rat Fa- Since Rat Fa is 
trivially closed under product, it follows from Corollarv 13.41 that R{X) is rational and 
effectively computable, and so is K{X) = R{X) n {R{X))-'^. By Theorem [TTl the 
subgroup K{X) is finitely generated and the proof is complete. □ 

These results are related to the Sakarovitch conjecture ||42l, which states that every 
rational subset of Fa must be either recognizable or disjunctive: a subset X of a monoid 
M is disjunctive if it has trivial syntactic congruence, or equivalently, if any morphism 
if : M —i' M' recognizing X is necessarily injective. 

In the group case, it follows easily from the proof of the direct implication of Lemma 
[33]that the projection G ^ G/N recognizes X CGif and only if TV C N{X). Thus X 
is disjunctive if and only if N{X) is the trivial subgroup. 

The Sakarovitch conjecture was first proved in [451, but once again we foUow the 
shorter alternative proof from B8l : 

Theorem 3.7 (Senizergues). A rational subset of Fa is either recognizable or disjunctive. 

Proof. Since the only subgroups of Z are the trivial subgroup and finite index subgroups, 
we may assume that > 1. 

Take X G Rati^^^. By the proof of Proposition [32] the subgroup K{X) is finitely 
generated. In view of Lemma [331 we may assume that K{X) is not a finite index sub- 
group. Thus S{K{X)) is not complete by Proposition l2.8l Let go denote the basepoint of 
S(K{X)). Since S{K{X)) is not complete, qq -uis undefined for some reduced word u. 

Let w be an arbitrary nonempty reduced word. We must show that w ^ -^(-'^')- 
Suppose otherwise. Since u, w are reduced and ^A > 1, there exist enough letters to 
make sure that there is some word v G Ra such that uvwv^^u^^ is reduced. Now 
w G N{X), hence uvwv^^u^^ G N{X) C K(X) by normality. Since uvwv~^u~^ is 
reduced, it follows from Proposition l2 .5 I that uvwv^^u^^ labels a loop at qo in S{K{X)), 
contradicting qo ■ u being undefined. Thus w ^ N{X) and so N{X) = 1. Therefore X 
is disjunctive as required. □ 



3.4 Beyond free groups 

Let TT : Fa -» G be a morphism onto a group G. We consider the word problem sub- 
monoid of a group G, defined as 

W^{G) = (7r0)-^(l). (3.3) 

Proposition 3.8. The language (G) is rational if and only ifG is finite. 

Proof. If G is finite, it is easy to check that W^(G) is rational by viewing the Cayley 
graph of G (see il24.1l i as an automaton. Conversely, if W^(G) is rational, then 7r~^(l) 
is a finitely generated normal subgroup of Fa, either finite index or trivial by the proof 
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of Theorem 13. 71 It is well known that the Dyck language Da = 6 ^(1) is not rational 
if #A > 0, thus it follows easily that 7r^^(l) has finite index and therefore G must be 
finite. □ 

How about groups with context-free W-„{G)1 A celebrated result by Muller and 
Schupp |33 1, with a contribution by Dunwoody 1J_3J, relates them to virtually free groups: 
these are groups with a free subgroup of finite index. 

As usual, we focus on the case of G being finitely generated. We claim that G has a 
normal free subgroup Fa of finite index, with A finite. Indeed, letting F be a finite-index 
free subgroup of G, it suffices to take F' = Plgec 9^9^^- Since F has finite index, so 
does F', see the proof of Lemma 13. 5 1 Taking a morphism vr : Fb G with B finite, we 
get from Corollary 12 . 91 that tt^^ (F') ^ /.^ Fb is finitely generated, so F' is itself finitely 
generated. Finally, F' is a subgroup of F, so F' is still free by Theorem 12. 71 and we can 
write F' ^ Fa- 

We may therefore decompose G as a finite disjoint union of the form 

G ^ FAboLlFAbiU---UFAb^, with &o = 1- (3.4) 



Theorem 3.9 (Muller & Schupp). The language WTr{G) is context-free if and only if G 
is virtually free. 

Sketch of proof. If G is virtually free, the rewriting system implicit in (13.4b provides a 
rational transduction between W7r(G) and Da- 

The converse implication can be proved by arguing geometrical properties of the Cay- 
ley graph of G such as in Chapter|24l briefly said, one deduces from the context-freeness 
of W-rriG) that the Cayley graph of G is close (more precisely, quasi-isometric) to a 
tree. □ 

It follows that virtually free groups have decidable word problem. In Chapter|24l we 
shall discuss the word problem for more general classes of groups using other techniques. 

Grunschlag proved that every rational (respectively recognizable) subset of a virtually 
free group G decomposed as in (13.41 ) admits a decomposition as a finite union Xq&o U 
■ • -UXmbni, where the are rational (respectively recognizable) subsets of Fa, see |18,]. 
Thus basic results such as Corollarv 13.41 or Proposition 13 .61 can be extended to virtually 
free groups (see lfT8ll47]| ). Similar generalizations can be obtained for free abelian groups 
of finite rank Il47l . 

The fact that the strong properties of Corollary 13.41 do hold for both free groups and 
free abelian groups suggests considering the case of graph groups (also known as free par- 
tially abelian groups or right angled Artin groups), where we admit partial commutation 
between letters. 

An independence graph is a finite undirected graph [A, I) with no loops, that is, / is a 
symmetric anti-reflexive relation on A. The graph group G(A, /) is the quotient Fa/ ~, 
where ~ denotes the congruence generated by the relation 

{{ahM) I {a,h) G /}. 

On both extremes, we have Fa — G{A, 0) and the free abelian group on A, which cor- 
responds to the complete graph on A. These turn out to be particular cases of transitive 
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forests. We can say that {A, I) is a transitive forest if it has no induced subgraph of either 
of the following forms: 

• • 

• • • • • • 

C4 P4 

We recall that an induced subgraph of {A, I) is formed by a subset of vertices A' A 
and all the edges in / connecting vertices from A'. 

The following difficult theorem, a group-theoretic version of a result on trace monoids 
by Aalbersberg and Hoogeboom |T|, was proved in ||231 : 

Theorem 3.10 (Lohrey & Steinberg). Let {A, I) be an independence graph. Then G{A, I) 
has decidable rational subset membership problem if and only if [A, I) is a transitive for- 
est. 

They also proved that these conditions are equivalent to decidability of the member- 
ship problem for finitely generated submonoids. Such a 'bad' G{A, I) gives an example 
of a finitely presented group with a decidable generalized word problem that does not 
have a decidable membership problem for finitely generated submonoids. 

It follows from Theorem 13 . 1 01 that any group containing a direct product of two free 
monoids has undecidable rational subset membership problem, a fact that can be directly 
deduced from the undecidability of the Post correspondence problem. 

Other positive results on rational subsets have been obtained for graphs of groups, 
HNN extensions and amalgamated free products by Kambites, Silva and Steinberg |[T9l . 
or Lohrey and Senizergues |22|. Lohrey and Steinberg proved recently that the rational 
subset membership problem is recursively equivalent to the finitely generated submonoid 
membership problem for groups with two or more ends ll24l . 

With respect to closure under complement, Lohrey and Senizergues fTT\ proved that 
the class of groups for which the rational subsets form a boolean algebra is closed under 
HNN extension and amalgamated products over finite groups. 

On the negative side, Bazhenova proved that rational subsets of finitely generated 
nilpotent groups do not form a boolean algebra, unless the group is virtually abelian [3J. 
Moreover, Roman'kov proved in 11411 . via a reduction from Hilbert's 10th problem, that 
the rational subset membership problem is undecidable for free nilpotent groups of any 
class ^ 2 of sufficiently large rank. 

Last but not least, we should mention that Stallings' construction was successfully 
generalized to prove results on both graph groups (by Kapovich, Miasnikov and Weid- 
mann li21J ') and amalgamated free products of finite groups (by Markus-Epstein [29J ). 

3.5 Rational solution sets and rational constraints 

In this final subsection we make a brief incursion in the brave new world of rational 
constraints. Rational subsets provide group theorists with two main assets: 
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• A concept which generalizes finite generation for subgroups and is much more fit 
to stand most induction procedures. 

• A systematic way of looking for solutions of the right type in the context of equa- 
tions of many sorts. 

This second feature leads us to the notion of rational constraint, when we restrict the set 
of potential solutions to some rational subset. And there is a particular combination of 
circumstances that can ensure the success of this strategy: if Rat G is closed under inter- 
section and we can prove that the solution set of problem P is an effectively computable 
rational subset of G, then we can solve problem P with any rational constraint. 

An early example is the adaptation by Margolis and Meakin of Rabin's language and 
Rabin's tree theorem to free groups, where first-order formulae provide rational solution 
sets (TT\. The logic language considered here is meant to be applied to words, seen as 
models, and consists basically of unary predicates that associate letters to positions in 
each word, as well as a binary predicate for position ordering. Margolis and Meakin used 
this construction to solve problems in combinatorial inverse semigroup theory ||27| . 

Diekert, Gutierrez and Hagenah proved that the existential theory of systems of equa- 
tions with rational constraints is solvable over a free group IfTTI . Working basically on 
a free monoid with involution, and adapting Plandowski's approach (35] in the process, 
they extended the classical result of Makanin [25.1 to include rational constraints, with 
much lower complexity as well. 

The proof of this deep result is well out of scope here, but its potential applications 
are immense. Group theorists are only starting to discover its full strength. 

The results in Il22ll can be used to extend the existential theory of equations with ra- 
tional constraints to virtually free groups, a result that follows also from Dahmani and 
Guirardel's recent paper on equations over hyperbolic groups with quasi-convex rational 
constraints ifTOl . Equations over graph groups with a restricted class of rational constraints 
were also successfully considered by Diekert and Lohrey 1 12 |. 

A somewhat exotic example of computation of a rational solution set arises in the 
problem of determining which automorphisms of F2 (if any) carry a given word into a 
given finitely generated subgroup. The full solution set is recognized by a finite automa- 
ton; its vertices are themselves structures named "finite truncated automata" ISTIl . 
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Abstract. This chapter is devoted to the study of rational subsets of groups, with particular em- 
phasis on the automata-theoretic approach to finitely generated subgroups of free groups. Indeed, 
Stallings' construction, associating a finite inverse automaton with every such subgroup, inaugu- 
rated a complete rewriting of free group algorithmics, with coimections to other fields such as 
topology or dynamics. 

Another important vector in the chapter is the fundamental Benois' Theorem, characterizing 
rational subsets of free groups. The theorem and its consequences really explain why language 
theory can be successfully applied to the study of free groups. Rational subsets of (free) groups can 
play a major role in proving statements (a priori unrelated to the notion of rationality) by induction. 
The chapter also includes related results for more general classes of groups, such as virtually free 
groups or graph groups. 
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